{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# @title Copyright & License (click to expand)\n",
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsv4jGuU89rX"
      },
      "source": [
        "# Vertex Model Monitoring\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/raw/master/notebooks/official/model_monitoring/model_monitoring.ipynb\"\n",
        "      <img src=\"https://cloud.google.com/images/products/ai/ai-solutions-icon.svg\" alt=\"Google Cloud Notebooks\"> Open in GCP Notebooks\n",
        "    </a>\n",
        "  </td> \n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
        "        <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lA32H1oKGgpf"
      },
      "source": [
        "## Overview"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t6Cd51FkG09E"
      },
      "source": [
        "### What is Model Monitoring?\n",
        "\n",
        "Modern applications rely on a well established set of capabilities to monitor the health of their services. Examples include:\n",
        "\n",
        "* software versioning\n",
        "* rigorous deployment processes\n",
        "* event logging\n",
        "* alerting/notication of situations requiring intervention\n",
        "* on-demand and automated diagnostic tracing\n",
        "* automated performance and functional testing\n",
        "\n",
        "You should be able to manage your ML services with the same degree of power and flexibility with which you can manage your applications. That's what MLOps is all about - managing ML services with the best practices Google and the broader computing industry have learned from generations of experience deploying well engineered, reliable, and scalable services.\n",
        "\n",
        "Model monitoring is only one piece of the ML Ops puzzle - it helps answer the following questions:\n",
        "\n",
        "* How well do recent service requests match the training data used to build your model? This is called **training-serving skew**.\n",
        "* How significantly are service requests evolving over time? This is called **drift detection**.\n",
        "\n",
        "If production traffic differs from  training data, or varies substantially over time, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that **you can anticipate problems before they affect your customer experiences or your revenue streams**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yG7FcXWKHOhC"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this notebook, you will learn how to... \n",
        "\n",
        "* deploy a pre-trained model\n",
        "* configure model monitoring\n",
        "* generate some artificial traffic\n",
        "* understand how to interpret the statistics, visualizations, other data reported by the model monitoring feature"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8yVpQt-JHKPF"
      },
      "source": [
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertext AI\n",
        "* BigQuery\n",
        "\n",
        "Learn about [Vertext AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "### The example model\n",
        "\n",
        "The model you'll use in this notebook is based on [this blog post](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). The idea behind this model is that your company has extensive log data describing how your game users have interacted with the site. The raw data contains the following categories of information:\n",
        "\n",
        "- identity - unique player identitity numbers\n",
        "- demographic features - information about the player, such as the geographic region in which a player is located\n",
        "- behavioral features - counts of the number of times a  player has triggered certain game events, such as reaching a new level\n",
        "- churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player.\n",
        "\n",
        "The blog article referenced above explains how to use BigQuery to store the raw data, pre-process it for use in machine learning, and train a model. Because this notebook focuses on model monitoring, rather than training models, you're going to reuse a pre-trained version of this model, which has been exported to Google Cloud Storage. In the next section, you will setup your environment and import this model into your own project."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ze4-nDLfK4pw"
      },
      "source": [
        "## Before you begin"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Setup your dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "020040f91150"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
        "USER_FLAG = \"\"\n",
        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    USER_FLAG = \"--user\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UcwUlnMnfo3i"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "import IPython\n",
        "\n",
        "assert sys.version_info.major == 3, \"This notebook requires Python 3.\"\n",
        "\n",
        "# Install Python package dependencies.\n",
        "print(\"Installing TensorFlow 2.4.1 and TensorFlow Data Validation (TFDV)\")\n",
        "! pip3 install {USER_FLAG} --quiet --upgrade tensorflow==2.4.1 tensorflow_data_validation[visualization]\n",
        "! pip3 install {USER_FLAG} --quiet --upgrade google-api-python-client google-auth-oauthlib google-auth-httplib2 oauth2client requests\n",
        "! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-aiplatform\n",
        "! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-storage==1.32.0\n",
        "\n",
        "# Automatically restart kernel after installing new packages.\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    print(\"Restarting kernel...\")\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)\n",
        "    print(\"Done.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8rv5wnmA6BTd"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import random\n",
        "import sys\n",
        "import time\n",
        "\n",
        "# Import required packages.\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
        "\n",
        "1. You'll use the *gcloud* command throughout this notebook. In the following cell, enter your project name and run the cell to authenticate yourself with the Google Cloud and initialize your *gcloud* configuration settings.\n",
        "\n",
        "**For this lab, we're going to use region us-central1 for all our resources (BigQuery training data, Cloud Storage bucket, model and endpoint locations, etc.). Those resources can be deployed in other regions, as long as they're consistently co-located, but we're going to use one fixed region to keep things as simple and error free as possible.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "wxiE6dEWOFm3"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "REGION = \"us-central1\"\n",
        "SUFFIX = \"aiplatform.googleapis.com\"\n",
        "API_ENDPOINT = f\"{REGION}-{SUFFIX}\"\n",
        "PREDICT_API_ENDPOINT = f\"{REGION}-prediction-{SUFFIX}\"\n",
        "if os.getenv(\"IS_TESTING\"):\n",
        "    !gcloud --quiet components install beta\n",
        "    !gcloud --quiet components update\n",
        "!gcloud config set project $PROJECT_ID\n",
        "!gcloud config set ai/region $REGION"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sECqTau7Oh6M"
      },
      "source": [
        "### Login to your Google Cloud account and enable AI services"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C6H1vZYjvT6w"
      },
      "outputs": [],
      "source": [
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "# If on Google Cloud Notebooks, then don't execute this code\n",
        "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    if \"google.colab\" in sys.modules:\n",
        "        from google.colab import auth as google_auth\n",
        "\n",
        "        google_auth.authenticate_user()\n",
        "\n",
        "    # If you are running this notebook locally, replace the string below with the\n",
        "    # path to your service account key and run this cell to authenticate your GCP\n",
        "    # account.\n",
        "    elif not os.getenv(\"IS_TESTING\"):\n",
        "        %env GOOGLE_APPLICATION_CREDENTIALS ''\n",
        "\n",
        "!gcloud services enable aiplatform.googleapis.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "btZeLzqQ7pXc"
      },
      "source": [
        "### Define some helper functions\n",
        "\n",
        "Run the following cell to define some utility functions used throughout this notebook. Although these functions are not critical to understand the main concepts, feel free to expand the  cell if you're curious or want to dive deeper into how some of your API requests are made."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yhDFSB2YDvfT"
      },
      "outputs": [],
      "source": [
        "# @title Utility functions\n",
        "import copy\n",
        "import os\n",
        "\n",
        "from google.cloud.aiplatform_v1beta1.services.endpoint_service import \\\n",
        "    EndpointServiceClient\n",
        "from google.cloud.aiplatform_v1beta1.services.job_service import \\\n",
        "    JobServiceClient\n",
        "from google.cloud.aiplatform_v1beta1.services.prediction_service import \\\n",
        "    PredictionServiceClient\n",
        "from google.cloud.aiplatform_v1beta1.types.io import BigQuerySource\n",
        "from google.cloud.aiplatform_v1beta1.types.model_deployment_monitoring_job import (\n",
        "    ModelDeploymentMonitoringJob, ModelDeploymentMonitoringObjectiveConfig,\n",
        "    ModelDeploymentMonitoringScheduleConfig)\n",
        "from google.cloud.aiplatform_v1beta1.types.model_monitoring import (\n",
        "    ModelMonitoringAlertConfig, ModelMonitoringObjectiveConfig,\n",
        "    SamplingStrategy, ThresholdConfig)\n",
        "from google.cloud.aiplatform_v1beta1.types.prediction_service import \\\n",
        "    PredictRequest\n",
        "from google.protobuf import json_format\n",
        "from google.protobuf.duration_pb2 import Duration\n",
        "from google.protobuf.struct_pb2 import Value\n",
        "\n",
        "DEFAULT_THRESHOLD_VALUE = 0.001\n",
        "\n",
        "\n",
        "def create_monitoring_job(objective_configs):\n",
        "    # Create sampling configuration.\n",
        "    random_sampling = SamplingStrategy.RandomSampleConfig(sample_rate=LOG_SAMPLE_RATE)\n",
        "    sampling_config = SamplingStrategy(random_sample_config=random_sampling)\n",
        "\n",
        "    # Create schedule configuration.\n",
        "    duration = Duration(seconds=MONITOR_INTERVAL)\n",
        "    schedule_config = ModelDeploymentMonitoringScheduleConfig(monitor_interval=duration)\n",
        "\n",
        "    # Create alerting configuration.\n",
        "    emails = [USER_EMAIL]\n",
        "    email_config = ModelMonitoringAlertConfig.EmailAlertConfig(user_emails=emails)\n",
        "    alerting_config = ModelMonitoringAlertConfig(email_alert_config=email_config)\n",
        "\n",
        "    # Create the monitoring job.\n",
        "    endpoint = f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{ENDPOINT_ID}\"\n",
        "    predict_schema = \"\"\n",
        "    analysis_schema = \"\"\n",
        "    job = ModelDeploymentMonitoringJob(\n",
        "        display_name=JOB_NAME,\n",
        "        endpoint=endpoint,\n",
        "        model_deployment_monitoring_objective_configs=objective_configs,\n",
        "        logging_sampling_strategy=sampling_config,\n",
        "        model_deployment_monitoring_schedule_config=schedule_config,\n",
        "        model_monitoring_alert_config=alerting_config,\n",
        "        predict_instance_schema_uri=predict_schema,\n",
        "        analysis_instance_schema_uri=analysis_schema,\n",
        "    )\n",
        "    options = dict(api_endpoint=API_ENDPOINT)\n",
        "    client = JobServiceClient(client_options=options)\n",
        "    parent = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n",
        "    response = client.create_model_deployment_monitoring_job(\n",
        "        parent=parent, model_deployment_monitoring_job=job\n",
        "    )\n",
        "    print(\"Created monitoring job:\")\n",
        "    print(response)\n",
        "    return response\n",
        "\n",
        "\n",
        "def get_thresholds(default_thresholds, custom_thresholds):\n",
        "    thresholds = {}\n",
        "    default_threshold = ThresholdConfig(value=DEFAULT_THRESHOLD_VALUE)\n",
        "    for feature in default_thresholds.split(\",\"):\n",
        "        feature = feature.strip()\n",
        "        thresholds[feature] = default_threshold\n",
        "    for custom_threshold in custom_thresholds.split(\",\"):\n",
        "        pair = custom_threshold.split(\":\")\n",
        "        if len(pair) != 2:\n",
        "            print(f\"Invalid custom skew threshold: {custom_threshold}\")\n",
        "            return\n",
        "        feature, value = pair\n",
        "        thresholds[feature] = ThresholdConfig(value=float(value))\n",
        "    return thresholds\n",
        "\n",
        "\n",
        "def get_deployed_model_ids(endpoint_id):\n",
        "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
        "    client = EndpointServiceClient(client_options=client_options)\n",
        "    parent = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n",
        "    response = client.get_endpoint(name=f\"{parent}/endpoints/{endpoint_id}\")\n",
        "    model_ids = []\n",
        "    for model in response.deployed_models:\n",
        "        model_ids.append(model.id)\n",
        "    return model_ids\n",
        "\n",
        "\n",
        "def set_objectives(model_ids, objective_template):\n",
        "    # Use the same objective config for all models.\n",
        "    objective_configs = []\n",
        "    for model_id in model_ids:\n",
        "        objective_config = copy.deepcopy(objective_template)\n",
        "        objective_config.deployed_model_id = model_id\n",
        "        objective_configs.append(objective_config)\n",
        "    return objective_configs\n",
        "\n",
        "\n",
        "def send_predict_request(endpoint, input):\n",
        "    client_options = {\"api_endpoint\": PREDICT_API_ENDPOINT}\n",
        "    client = PredictionServiceClient(client_options=client_options)\n",
        "    params = {}\n",
        "    params = json_format.ParseDict(params, Value())\n",
        "    request = PredictRequest(endpoint=endpoint, parameters=params)\n",
        "    inputs = [json_format.ParseDict(input, Value())]\n",
        "    request.instances.extend(inputs)\n",
        "    response = client.predict(request)\n",
        "    return response\n",
        "\n",
        "\n",
        "def list_monitoring_jobs():\n",
        "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
        "    parent = f\"projects/{PROJECT_ID}/locations/us-central1\"\n",
        "    client = JobServiceClient(client_options=client_options)\n",
        "    response = client.list_model_deployment_monitoring_jobs(parent=parent)\n",
        "    print(response)\n",
        "\n",
        "\n",
        "def pause_monitoring_job(job):\n",
        "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
        "    client = JobServiceClient(client_options=client_options)\n",
        "    response = client.pause_model_deployment_monitoring_job(name=job)\n",
        "    print(response)\n",
        "\n",
        "\n",
        "def delete_monitoring_job(job):\n",
        "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
        "    client = JobServiceClient(client_options=client_options)\n",
        "    response = client.delete_model_deployment_monitoring_job(name=job)\n",
        "    print(response)\n",
        "\n",
        "\n",
        "# Sampling distributions for categorical features...\n",
        "DAYOFWEEK = {1: 1040, 2: 1223, 3: 1352, 4: 1217, 5: 1078, 6: 1011, 7: 1110}\n",
        "LANGUAGE = {\n",
        "    \"en-us\": 4807,\n",
        "    \"en-gb\": 678,\n",
        "    \"ja-jp\": 419,\n",
        "    \"en-au\": 310,\n",
        "    \"en-ca\": 299,\n",
        "    \"de-de\": 147,\n",
        "    \"en-in\": 130,\n",
        "    \"en\": 127,\n",
        "    \"fr-fr\": 94,\n",
        "    \"pt-br\": 81,\n",
        "    \"es-us\": 65,\n",
        "    \"zh-tw\": 64,\n",
        "    \"zh-hans-cn\": 55,\n",
        "    \"es-mx\": 53,\n",
        "    \"nl-nl\": 37,\n",
        "    \"fr-ca\": 34,\n",
        "    \"en-za\": 29,\n",
        "    \"vi-vn\": 29,\n",
        "    \"en-nz\": 29,\n",
        "    \"es-es\": 25,\n",
        "}\n",
        "OS = {\"IOS\": 3980, \"ANDROID\": 3798, \"null\": 253}\n",
        "MONTH = {6: 3125, 7: 1838, 8: 1276, 9: 1718, 10: 74}\n",
        "COUNTRY = {\n",
        "    \"United States\": 4395,\n",
        "    \"India\": 486,\n",
        "    \"Japan\": 450,\n",
        "    \"Canada\": 354,\n",
        "    \"Australia\": 327,\n",
        "    \"United Kingdom\": 303,\n",
        "    \"Germany\": 144,\n",
        "    \"Mexico\": 102,\n",
        "    \"France\": 97,\n",
        "    \"Brazil\": 93,\n",
        "    \"Taiwan\": 72,\n",
        "    \"China\": 65,\n",
        "    \"Saudi Arabia\": 49,\n",
        "    \"Pakistan\": 48,\n",
        "    \"Egypt\": 46,\n",
        "    \"Netherlands\": 45,\n",
        "    \"Vietnam\": 42,\n",
        "    \"Philippines\": 39,\n",
        "    \"South Africa\": 38,\n",
        "}\n",
        "\n",
        "# Means and standard deviations for numerical features...\n",
        "MEAN_SD = {\n",
        "    \"julianday\": (204.6, 34.7),\n",
        "    \"cnt_user_engagement\": (30.8, 53.2),\n",
        "    \"cnt_level_start_quickplay\": (7.8, 28.9),\n",
        "    \"cnt_level_end_quickplay\": (5.0, 16.4),\n",
        "    \"cnt_level_complete_quickplay\": (2.1, 9.9),\n",
        "    \"cnt_level_reset_quickplay\": (2.0, 19.6),\n",
        "    \"cnt_post_score\": (4.9, 13.8),\n",
        "    \"cnt_spend_virtual_currency\": (0.4, 1.8),\n",
        "    \"cnt_ad_reward\": (0.1, 0.6),\n",
        "    \"cnt_challenge_a_friend\": (0.0, 0.3),\n",
        "    \"cnt_completed_5_levels\": (0.1, 0.4),\n",
        "    \"cnt_use_extra_steps\": (0.4, 1.7),\n",
        "}\n",
        "\n",
        "DEFAULT_INPUT = {\n",
        "    \"cnt_ad_reward\": 0,\n",
        "    \"cnt_challenge_a_friend\": 0,\n",
        "    \"cnt_completed_5_levels\": 1,\n",
        "    \"cnt_level_complete_quickplay\": 3,\n",
        "    \"cnt_level_end_quickplay\": 5,\n",
        "    \"cnt_level_reset_quickplay\": 2,\n",
        "    \"cnt_level_start_quickplay\": 6,\n",
        "    \"cnt_post_score\": 34,\n",
        "    \"cnt_spend_virtual_currency\": 0,\n",
        "    \"cnt_use_extra_steps\": 0,\n",
        "    \"cnt_user_engagement\": 120,\n",
        "    \"country\": \"Denmark\",\n",
        "    \"dayofweek\": 3,\n",
        "    \"julianday\": 254,\n",
        "    \"language\": \"da-dk\",\n",
        "    \"month\": 9,\n",
        "    \"operating_system\": \"IOS\",\n",
        "    \"user_pseudo_id\": \"104B0770BAE16E8B53DF330C95881893\",\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lAOk8UqvCL0S"
      },
      "source": [
        "## Import your model\n",
        "\n",
        "The churn propensity model you'll be using in this notebook has been trained in BigQuery ML and exported to a Google Cloud Storage bucket. This illustrates how you can easily export a trained model and move a model from one cloud service to another. \n",
        "\n",
        "Run the next cell to import this model into your project. **If you've already imported your model, you can skip this step.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Se5DxJoK3AMz"
      },
      "outputs": [],
      "source": [
        "MODEL_NAME = \"churn\"\n",
        "IMAGE = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-4:latest\"\n",
        "ARTIFACT = \"gs://mco-mm/churn\"\n",
        "output = !gcloud --quiet beta ai models upload --container-image-uri=$IMAGE --artifact-uri=$ARTIFACT --display-name=$MODEL_NAME --format=\"value(model)\"\n",
        "print(\"model output: \", output)\n",
        "MODEL_ID = output[1].split(\"/\")[-1]\n",
        "print(f\"Model {MODEL_NAME}/{MODEL_ID} created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L1HzDTh-JRsr"
      },
      "source": [
        "## Deploy your endpoint\n",
        "\n",
        "Now that you've imported your model into your project, you need to create an endpoint to serve your model. An endpoint can be thought of as a channel through which your model provides prediction services. Once established, you'll be able to make prediction requests on your model via the public internet.  Your endpoint is also serverless, in the sense that Google ensures high availability by reducing single points of failure, and scalability by dynamically allocating resources to meet the demand for your service. In this way, you are able to focus on your model quality, and freed from adminstrative and infrastructure concerns.\n",
        "\n",
        "Run the next cell to deploy your model to an endpoint. **This will take about ten minutes to complete. If you've already deployed a model to an endpoint, you can reuse your endpoint by running the cell after the next one.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M4nHQTVh7PIi"
      },
      "outputs": [],
      "source": [
        "ENDPOINT_NAME = \"churn\"\n",
        "output = !gcloud --quiet beta ai endpoints create --display-name=$ENDPOINT_NAME --format=\"value(name)\"\n",
        "print(\"endpoint output: \", output)\n",
        "ENDPOINT = output[-1]\n",
        "ENDPOINT_ID = ENDPOINT.split(\"/\")[-1]\n",
        "output = !gcloud --quiet beta ai endpoints deploy-model $ENDPOINT_ID --display-name=$ENDPOINT_NAME --model=$MODEL_ID --traffic-split=\"0=100\"\n",
        "DEPLOYED_MODEL_ID = output[1].split()[-1][:-1]\n",
        "print(\n",
        "    f\"Model {MODEL_NAME}/{MODEL_ID}/{DEPLOYED_MODEL_ID} deployed to Endpoint {ENDPOINT_NAME}/{ENDPOINT_ID}/{ENDPOINT}.\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QzZS5piBeBFI"
      },
      "outputs": [],
      "source": [
        "# @title Run this cell only if you want to reuse an existing endpoint.\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    ENDPOINT_ID = \"\"  # @param {type:\"string\"}\n",
        "    ENDPOINT = f\"projects/mco-mm/locations/us-central1/endpoints/{ENDPOINT_ID}\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NKsA_lfl9Ryw"
      },
      "source": [
        "## Run a prediction test\n",
        "\n",
        "Now that you have imported a model and deployed that model to an endpoint, you are ready to verify that it's working. Run the next cell to send a test prediction request. If everything works as expected, you should receive a response encoded in a text representation called JSON.\n",
        "\n",
        "**Try this now by running the next cell and examine the results.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QNEb7fDJ9NXc"
      },
      "outputs": [],
      "source": [
        "import pprint as pp\n",
        "\n",
        "print(ENDPOINT)\n",
        "print(\"request:\")\n",
        "pp.pprint(DEFAULT_INPUT)\n",
        "try:\n",
        "    resp = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n",
        "    print(\"response\")\n",
        "    pp.pprint(resp)\n",
        "except Exception:\n",
        "    print(\"prediction request failed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lN6ADMJqeWtR"
      },
      "source": [
        "Taking a closer look at the results, we see the following elements:\n",
        "\n",
        "- **churned_values** - a set of possible values (0 and 1) for the target field\n",
        "- **churned_probs** - a corresponding set of probabilities for each possible target field value (5x10^-40 and 1.0, respectively)\n",
        "- **predicted_churn** - based on the probabilities, the predicted value of the target field (1)\n",
        "\n",
        "This response encodes the model's prediction in a format that is readily digestible by software, which makes this service ideal for automated use by an application."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8rF5iLuXCT7i"
      },
      "source": [
        "## Start your monitoring job\n",
        "\n",
        "Now that you've created an endpoint to serve prediction requests on your model, you're ready to start a monitoring job to keep an eye on model quality and to alert you if and when input begins to deviate in way that may impact your model's prediction quality.\n",
        "\n",
        "In this section, you will configure and create a model monitoring job based on the churn propensity model you imported from BigQuery ML."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wW2gLBQ3Zkhq"
      },
      "source": [
        "### Configure the following fields:\n",
        "\n",
        "1. Log sample rate - Your prediction requests and responses are logged to BigQuery tables, which are automatically created when you create a monitoring job. This parameter specifies the desired logging frequency for those tables.\n",
        "1. Monitor interval - the  time window over which to analyze your data and report anomalies. The minimum window is one hour (3600 seconds).\n",
        "1. Target field - the prediction target column name in training dataset.\n",
        "1. Skew detection threshold - the skew threshold for each feature you want to monitor.\n",
        "1. Prediction drift threshold - the drift threshold for each feature you want to monitor."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "plpASmM2YIVO"
      },
      "outputs": [],
      "source": [
        "USER_EMAIL = \"\"  # @param {type:\"string\"}\n",
        "JOB_NAME = \"churn\"\n",
        "\n",
        "# Sampling rate (optional, default=.8)\n",
        "LOG_SAMPLE_RATE = 0.8  # @param {type:\"number\"}\n",
        "\n",
        "# Monitoring Interval in seconds (optional, default=3600).\n",
        "MONITOR_INTERVAL = 3600  # @param {type:\"number\"}\n",
        "\n",
        "# URI to training dataset.\n",
        "DATASET_BQ_URI = \"bq://mco-mm.bqmlga4.train\"  # @param {type:\"string\"}\n",
        "# Prediction target column name in training dataset.\n",
        "TARGET = \"churned\"\n",
        "\n",
        "# Skew and drift thresholds.\n",
        "SKEW_DEFAULT_THRESHOLDS = \"country,language\"  # @param {type:\"string\"}\n",
        "SKEW_CUSTOM_THRESHOLDS = \"cnt_user_engagement:.5\"  # @param {type:\"string\"}\n",
        "DRIFT_DEFAULT_THRESHOLDS = \"country,language\"  # @param {type:\"string\"}\n",
        "DRIFT_CUSTOM_THRESHOLDS = \"cnt_user_engagement:.5\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjVSViZR-dP2"
      },
      "source": [
        "### Create your monitoring job\n",
        "\n",
        "The following code uses the Google Python client library to translate your configuration settings into a programmatic request to start a model monitoring job. Instantiating a monitoring job can take some time. If everything looks good with your request, you'll get a successful API response. Then, you'll need to check your email to receive a notification that the job is running."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-62TYm2iYv3K"
      },
      "outputs": [],
      "source": [
        "skew_thresholds = get_thresholds(SKEW_DEFAULT_THRESHOLDS, SKEW_CUSTOM_THRESHOLDS)\n",
        "drift_thresholds = get_thresholds(DRIFT_DEFAULT_THRESHOLDS, DRIFT_CUSTOM_THRESHOLDS)\n",
        "skew_config = ModelMonitoringObjectiveConfig.TrainingPredictionSkewDetectionConfig(\n",
        "    skew_thresholds=skew_thresholds\n",
        ")\n",
        "drift_config = ModelMonitoringObjectiveConfig.PredictionDriftDetectionConfig(\n",
        "    drift_thresholds=drift_thresholds\n",
        ")\n",
        "training_dataset = ModelMonitoringObjectiveConfig.TrainingDataset(target_field=TARGET)\n",
        "training_dataset.bigquery_source = BigQuerySource(input_uri=DATASET_BQ_URI)\n",
        "objective_config = ModelMonitoringObjectiveConfig(\n",
        "    training_dataset=training_dataset,\n",
        "    training_prediction_skew_detection_config=skew_config,\n",
        "    prediction_drift_detection_config=drift_config,\n",
        ")\n",
        "model_ids = get_deployed_model_ids(ENDPOINT_ID)\n",
        "objective_template = ModelDeploymentMonitoringObjectiveConfig(\n",
        "    objective_config=objective_config\n",
        ")\n",
        "objective_configs = set_objectives(model_ids, objective_template)\n",
        "\n",
        "monitoring_job = create_monitoring_job(objective_configs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OiwOVR4D_xhl"
      },
      "outputs": [],
      "source": [
        "# Run a prediction request to generate schema, if necessary.\n",
        "try:\n",
        "    _ = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n",
        "    print(\"prediction succeeded\")\n",
        "except Exception:\n",
        "    print(\"prediction failed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SaXYVFFslRru"
      },
      "source": [
        "After a minute or two, you should receive email at the address you configured above for USER_EMAIL. This email confirms successful deployment of your monitoring job. Here's a sample of what this email might look like:\n",
        "<br>\n",
        "<br>\n",
        "<img src=\"https://storage.googleapis.com/mco-general/img/mm6.png\" />\n",
        "<br>\n",
        "As your monitoring job collects data, measurements are stored in Google Cloud Storage and you are free to examine your data at any time. The circled path in the image above specifies the location of your measurements in Google Cloud Storage. Run the following cell to take a look at your measurements in Cloud Storage.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XV-vru2Pm1oX"
      },
      "outputs": [],
      "source": [
        "!gsutil ls gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/*/*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XgUwU0sDpUUD"
      },
      "source": [
        "You will notice the following components in these Cloud Storage paths:\n",
        "\n",
        "- **cloud-ai-platform-..** - This is a bucket created for you and assigned to capture your service's prediction data. Each monitoring job you create will trigger creation of a new folder in this bucket.\n",
        "- **[model_monitoring|instance_schemas]/job-..** - This is your unique monitoring job number, which you can see above in both the response to your job creation requesst and the email notification. \n",
        "- **instance_schemas/job-../analysis** - This is the monitoring jobs understanding and encoding of your training data's schema (field names, types, etc.).\n",
        "- **instance_schemas/job-../predict** - This is the first prediction made to your model after the current monitoring job was enabled.\n",
        "- **model_monitoring/job-../serving** - This folder is used to record data relevant to drift calculations. It contains measurement summaries for every hour your model serves traffic.\n",
        "- **model_monitoring/job-../training** - This folder is used to record data relevant to training-serving skew calculations. It contains an ongoing summary of prediction data relative to training data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8V2zo7-MMd7G"
      },
      "source": [
        "### You can create monitoring jobs with other user interfaces\n",
        "\n",
        "In the previous cells, you created a monitoring job using the Python client library. You can also use the *gcloud* command line tool to create a model monitoring job and, in the near future, you will be able to use the Cloud Console, as well for this function. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q106INuFCXKX"
      },
      "source": [
        "## Generate test data to trigger alerting\n",
        "\n",
        "Now you are ready to test the monitoring function. Run the following cell, which will generate fabricated test predictions designed to trigger the thresholds you specified above. It takes about five minutes to run this cell and at least an hour to assess and report anamolies in skew or drift so after running this cell, feel free to proceed with the notebook and you'll see how to examine the resulting alert later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "obZYLLAuKmG8"
      },
      "outputs": [],
      "source": [
        "def random_uid():\n",
        "    digits = [str(i) for i in range(10)] + [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"]\n",
        "    return \"\".join(random.choices(digits, k=32))\n",
        "\n",
        "\n",
        "def monitoring_test(count, sleep, perturb_num={}, perturb_cat={}):\n",
        "    # Use random sampling and mean/sd with gaussian distribution to model\n",
        "    # training data. Then modify sampling distros for two categorical features\n",
        "    # and mean/sd for two numerical features.\n",
        "    mean_sd = MEAN_SD.copy()\n",
        "    country = COUNTRY.copy()\n",
        "    for k, (mean_fn, sd_fn) in perturb_num.items():\n",
        "        orig_mean, orig_sd = MEAN_SD[k]\n",
        "        mean_sd[k] = (mean_fn(orig_mean), sd_fn(orig_sd))\n",
        "    for k, v in perturb_cat.items():\n",
        "        country[k] = v\n",
        "    for i in range(0, count):\n",
        "        input = DEFAULT_INPUT.copy()\n",
        "        input[\"user_pseudo_id\"] = str(random_uid())\n",
        "        input[\"country\"] = random.choices([*country], list(country.values()))[0]\n",
        "        input[\"dayofweek\"] = random.choices([*DAYOFWEEK], list(DAYOFWEEK.values()))[0]\n",
        "        input[\"language\"] = str(random.choices([*LANGUAGE], list(LANGUAGE.values()))[0])\n",
        "        input[\"operating_system\"] = str(random.choices([*OS], list(OS.values()))[0])\n",
        "        input[\"month\"] = random.choices([*MONTH], list(MONTH.values()))[0]\n",
        "        for key, (mean, sd) in mean_sd.items():\n",
        "            sample_val = round(float(np.random.normal(mean, sd, 1)))\n",
        "            val = max(sample_val, 0)\n",
        "            input[key] = val\n",
        "        print(f\"Sending prediction {i}\")\n",
        "        try:\n",
        "            send_predict_request(ENDPOINT, input)\n",
        "        except Exception:\n",
        "            print(\"prediction request failed\")\n",
        "        time.sleep(sleep)\n",
        "    print(\"Test Completed.\")\n",
        "\n",
        "\n",
        "test_time = 300\n",
        "tests_per_sec = 1\n",
        "sleep_time = 1 / tests_per_sec\n",
        "iterations = test_time * tests_per_sec\n",
        "perturb_num = {\"cnt_user_engagement\": (lambda x: x * 3, lambda x: x / 3)}\n",
        "perturb_cat = {\"Japan\": max(COUNTRY.values()) * 2}\n",
        "monitoring_test(iterations, sleep_time, perturb_num, perturb_cat)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bQohDTJgLQlW"
      },
      "source": [
        "## Interpret your results\n",
        "\n",
        "While waiting for your results, which, as noted, may take up to an hour, you can read ahead to get sense of the alerting experience."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uGPI92qbOFUR"
      },
      "source": [
        "### Here's what a sample email alert looks like...\n",
        "\n",
        "<img src=\"https://storage.googleapis.com/mco-general/img/mm7.png\" />\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HoaqsxpaRs1m"
      },
      "source": [
        "This email is warning you that the *cnt_user_engagement*, *country* and *language* feature values seen in production have skewed above your threshold between training and serving your model. It's also telling you that the *cnt_user_engagement* feature value is drifting significantly over time, again, as per your threshold specification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w4jVIVq4VzB_"
      },
      "source": [
        "### Monitoring results in the Cloud Console\n",
        "\n",
        "You can examine your model monitoring data from the Cloud Console. Below is a screenshot of those capabilities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2OdIMVBAPZi_"
      },
      "source": [
        "#### Monitoring Status\n",
        "\n",
        "<img src=\"https://storage.googleapis.com/mco-general/img/mm1.png\" />\n",
        "\n",
        "#### Monitoring Alerts\n",
        "\n",
        "<img src=\"https://storage.googleapis.com/mco-general/img/mm2.png\" />"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Clean up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TPP_ImwJDFJf"
      },
      "outputs": [],
      "source": [
        "# Delete endpoint resource\n",
        "!gcloud ai endpoints delete $ENDPOINT_NAME --quiet\n",
        "\n",
        "# Delete model resource\n",
        "!gcloud ai models delete $MODEL_NAME --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3Dh15h3-NoO"
      },
      "source": [
        "## Learn more about model monitoring\n",
        "\n",
        "**Congratulations!** You've now learned what model monitoring is, how to configure and enable it, and how to find and interpret the results. Check out the following resources to learn more about model monitoring and ML Ops.\n",
        "\n",
        "- [TensorFlow Data Validation](https://www.tensorflow.org/tfx/guide/tfdv)\n",
        "- [Data Understanding, Validation, and Monitoring At Scale](https://blog.tensorflow.org/2018/09/introducing-tensorflow-data-validation.html)\n",
        "- [Vertex Product Documentation](https://cloud.google.com/vertex)\n",
        "- [Model Monitoring Reference Docs](https://cloud.google.com/vertex/docs/reference)\n",
        "- [Model Monitoring blog article](https://cloud.google.com/blog/topics/developers-practitioners/monitor-models-training-serving-skew-vertex-ai)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "lA32H1oKGgpf"
      ],
      "name": "model_monitoring.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
